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Introduction 



There is renewed interest in the idea of individualized instruction, driven in part 
by advances in technology (e.g., ADL, 2006; IEEE LTSC, 2006), advances in assessment 
(e.g., NRC, 2001; Williamson, Behar, & Mislevy, 2006), and a persistent desire to 
increase the access, efficiency, and cost effectiveness of training and education (e.g., 
Eletcher, Tobias, & Wisher, 2006). While the idea of individualized instruction has 
existed for some time (Courtis, 1938), what is remarkable are the striking similarities of 
desired goals and methods between current research in training and education and 
work beginning almost a century ago (e.g., teaching machines [Pressey, 1926, 1927; 
Skinner, 1958; Thorndike, 1912], programmed instruction [Lumsdaine & Glaser, 1960; 
McDonald, Yanchar, & Osguthorpe, 2005], mastery learning [Bloom, 1968], domain- 
referenced [Baker, 1974; Hively, 1974; Hively, Patterson, & Page, 1968] and criterion- 
referenced testing [Glaser, 1963], CAI [Atkinson, 1968; Suppes & Morningstar, 1969], 
intrinsic programming [Crowder, 1960], hypertext [Engelbart, 1962]). What differs 
today, however, is the availability of technology to make practical many of the ideas 
central to individualizing instruction. Eirst, the means now exist to deliver tasks that 
can implement and streamline many of the capabilities that were cumbersome in the 
original formulations of programmed instruction, e.g., embedded and dynamic testing, 
immediate feedback, active participation during instruction, and instructional 
branching. All were recognized as important to increase learning, but their 
implementation was limited by the available technology of the time. Second and 
perhaps more important, advances in the science and technology of the assessment of 
student learning have enabled cost-effective embedded assessments supporting 
feedback, diagnosis, and selection and delivery of appropriate instructional parcels, a 
capability that promises to deliver the individualized instruction (Baker, 1997; Mislevy 
& Riconscente, 2005; NRC, 2001). 

In this paper, we describe our research and development efforts at individualizing 
instruction for low-knowledge students learning pre-algebra. To support this effort, our 
development was part engineering (draw on the strongest empirical evidence to design 
the instruction), part content development (developing instruction on several pre- 
algebra concepts), part research in instructional design (testing whether very brief 
exposure to pre-algebra instruction will result in learning at all), part assessment 
research (testing a novel "next step" assessment for rapid and precise diagnosis), and 
part technology development (testing automated reasoning techniques with Bayesian 
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networks). Our long-term goal is to develop an approach to support "faster, cheaper, 
and better" ways of delivering individual instruction in a distributed learning context. 

In this paper, we report on a subset of the work: (a) the instructional design, (b) the 
assessment design, and (c) a study on the effectiveness of very brief exposure to 
instruction. Implications for education, training, and online systems are discussed. 

Designing Instruction for Efficient Learning 

A basic assumption underlying our instructional design was that for instruction to 
be maximally effective, particularly when brief, the instructional design should 
incorporate the features known to promote learning. We drew extensively on the work 
related to multimedia learning and cognitive load, analogical reasoning, and feedback 
(e.g.. Black, Harrison, Lee, Marshall, & Wiliam, 2003; Catrambone, 1998; Chandler & 
Sweller, 1991; Kluger & DeNisi, 1996; Mayer, 2001, 2005b; Novick & Holyoak, 1991; 
Sweller, van Merrienboer, & Paas, 1998). Our goal was to implement in computer-based 
instruction the properties with strong empirical evidence of effectiveness to deliver 
extremely efficient instruction targeted to low-knowledge learners. 

Conceptual Instruction 

There were two broad instructional design objectives we set for the actual 
software. The first objective addressed the overall structure of the instruction— how 
should the to-be-learned information be structured to facilitate understanding in 
content that is typically "brittle" (application of math procedures under very specific 
conditions and tied to the surface form of the math expression)? We addressed this 
objective by providing instruction that conveyed both the concepts and procedures of 
the underlying math concept. Importantly, instruction was based on multiple examples 
applied across different math problems. The desired learning outcome was for students 
to understand that the same solution method or reasoning applied across problems that 
differed in surface features. Instruction was chunked into three areas: what (i.e., the 
goal of underlying the math procedure), how (i.e., the procedure), and why (i.e., an 
explanation of why the particular math concept or procedure "works"). 

We designed the instruction to emphasize the relational correspondences among 
examples. Gick and Holyoak (1983) found that performance on transfer items was 
improved if instruction made explicit the comparisons among examples to promote the 
recognition of similarities. One technique is to require learners to explicitly compare 
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examples for similarities (Catrambone & Holyoak, 1989; Cummins, 1992; Novick & 
Holy oak, 1991). Such explicit comparisons promote performance on transfer tasks by 
guiding learner's attention to the more abstract generalizations that might not be 
obvious in the surface features of the examples. This issue may be particularly 
important with low prior knowledge learners, where directing learners' attention to 
superficial but semantically related aspects of the problems appears key to learning the 
underlying structure (Catrambone, 1998). Figure 1 shows sample instruction of 
introducing the goal behind the concept of transformations. Transformations refers to the 
idea of isolating variables by using inverse operations. 



« + 2 = 36 .v-4 = 6 3r = 15 ^ = 12 

4 

|a + 2 -2 = 36 + 2-2| ,r-4 + 4-6 + 4 ( 3r + 3-15 + 3] T. 4 ^12.4 




Goal: Find the Value of the Variable 



Narration: What are we trying to do? Even though they all look different, we have the same 
goal. We're trying to find the value of the variable. We want to know what a, x, z, and y are 
equal to. 

Figure 1. Example of instruction that emphasizes the underlying math concept across different surface 
examples. The actual on-screen instruction unfolds in steps and the visual highlighting occurs in 

coordination with the narration. 



Multimedia-based Instruction 

The second instructional design objective addressed the delivery of the 
instruction— what techniques could be used to facilitate the communication of the 
content? We focused on techniques that specifically addressed limitations of human 
cognition (e.g., limited working memory capacity), that exploited human sensory 
channels (visual, auditory), that would be appropriate for math, and that were within 
the capabilities of the available technology. Where practical, we also adopted 
techniques with large effect sizes. We addressed the second objective by adopting many 
of the guidelines derived from research on multimedia and cognitive load (e.g., Mayer, 
2001, 2005a; Clark, Nguyen, & Sweller, 2006). Figure 2 shows an example of what the 
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screen looked like. Note that students could control pacing via the control buttons on 
the lower right of the screen. 




Figure 2. Sample screen shot illustrating (a) multiple examples, (b) cuing, (c) segmenting, and (d) split 

attention. 



Worked-example Practice 

Similar to the instruction, practice was with multiple examples with different 
surface forms but the same underlying math concept. The practice stage involved 
students "solving" three math problems in parallel using a "next step" approach. We 
attempted to implement a simplified worked example in three stages. First, students 
were required to identify the appropriate next step of solving the problems. Next, given 
that students were successful, they were asked to identify the expression (i.e., if one 
carried out the next step procedure, what is the resultant expression?). Finally, given 
that students were successful, they were required to identify the underlying math 
concept. Figure 3 shows a sample screen. Note that in the actual application, the 
practice is administered in three stages, and only the current and previous stages are 
shown on the screen. The student advances to the next stage only after getting all the 
questions correct on the current stage. 

The purpose of this approach was to provide practice that emphasized the 
important first steps involved in solving math problems, to expose students to math 
language usage when applying math operations, to provide students with practice 
applying a math concept on different surface forms of the problem, to have the technical 
means of capturing students' online problem solving processes, and to gather 
information on whether students commit common errors. 
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The use of the "next step" was inspired by the work of Kalyuga and Sweller (2004) 
and Kalyuga (2006). They hypothesized that what students wrote down as the first step 
in solving an equation reflected their level of expertise in solving equations. High 
correlations (rs between .7 and .9) were found between performance on specifying the 
first step of a solution and performance on specifying the entire solution. Their finding 
suggests that the first step is the critical event, particularly for low prior knowledge 
students. Thus our instruction attempted to support students in identifying the critical 
first step. 



Multiple Examples 



/ \ 

What can you do to ch^ge the expressiot^^hout changing itf? value? 



PART 1: What would your next step be in solving these problems? 
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3 

5 
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PART 2: Qick on the text box that reflects the result of the above operation. Click the Done With This Part** button for feedback. 







5x- 1 






5jt + 1 


,0-H 




5x-l 


Doni know 




DonT know 



3 5 




Click here 




5 5 




Stage 2: What is the 


5 * 5 




resuit of carrying 


3 5 


out the step? 






Oonl know 


Review Feedback | 





PART 3: Why can you do this? 

^ Identity property of addition I 

^ Identity property of multipltcalion 

Inverse property of addition • 

f* Inverse property of multiplication 
^ Equality property of addition 
Equality property of multiplication 
^ Associative property of addition 
Associative property of multiplication 
C* Commutative property of addition 
^ Commutative property of multiplication 
C" Distnbutive property 
Add. Sub. Mult .Dw 
Doni know 



Identity property of addition I 

Identity property of mutliplicatioo 
Inverse property of addition * 

f“ Inverse property of multiplication 
Equakty property of addition 
Equakty property of multiplication 
f* Associatrre property of addition 
C* Associatrue property of multiplication 
^ Commutative property of addition 
^ Commutative property of multiplication 
Distributive property 
f" Add. Sub . Mult . Orv 
^ Ooni know 



' loeniny propeny or aoanion 
C* Identity property of multiplK |||Q||h 



f' Inverse property of addition 
C* Inverse property of multiplic 
Equality property of additioi 
^ Equality property of muitipli 
C* Associative property of add 
Associative property of muf 
f* Commutative property of ac 
C* Commutative property of muttipiication 
C* Distnbutive property 
f* Add. Sub . Mult . Dfv 
^ Doni know 



Stage 3: What is tl 
underiying math 
concept? 



Done with This Part 



Figure 3. Screen shot showing the practice. 



Tailored Feedback 

After attempting to "solve" the problems at each stage, feedback was provided to 
students. The feedback included (a) knowledge of results [whether the student got the 
problem correct or incorrect], and (b) explanatory feedback that provided guidance to 
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learners on what they should focus on to solve the problem. These two techniques have 
been found to be effective feedback methods, particularly for learners with low 
knowledge of the domain (e.g., Azevedo & Bernard, 1995; Bangert-Drowns, Kulick, & 
Morgan, 1991; Black et al., 2003; Kluger & DeNisi, 1996). 

If the student specified an incorrect response, the explanatory feedback stated 
what was wrong with the response and a hint about what the student should think 
about to correctly solve the problem. If the student specified a correct response, the 
feedback explained why the response was correct. If the student specified "don't 
know," the feedback provided guidance on what the student should consider. Finally, 
access to segments of the original instruction was provided. Students could play the 
video that related to the goal, why, or how instructions that were relevant to the 
particular practice problem. In addition, students could view instruction on related 
common errors. Such feedback was provided for all three stages of the practice. The 
general timing of the feedback was based on findings from Kester, Kirschner, and van 
Merrienboer (2005), who found that procedural information presented prior to the 
practice task and explanatory feedback provided during the practice task led to the 
most efficient learning. 
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Figure 4. Sample feedback screen. 



Task Sequence 

The general format of instruction was an initial animation-based instruction 
followed by practice opportunities with tailored feedback. Figure 5 shows the flowchart 
of the task sequence. 
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Figure 5. Flow chart of general task sequence. There are 4 general stages: (a) initial instruction, (b) 
identifying the "next step" in solving a problem, (c) identifying the equation or expression that results 
from carrying ouf fhe sfep, and (d) identifying the common underlying math property. 



Summary 

Thus, our overall instructional design strategy was pragmatic and focused on 
addressing the question: To maximize the chances of learning, which instructional 
features have been shown to be very effective, relevant to math, and traceable to an 
empirical research base? Table 1 summarizes the design properties of the software, our 
implementation, and the research we based the property on. A fuller description is 
given in Appendix A. 
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Table 1 

Summary of Instructional Design Features Adopted 

Instructional design property Research base 

Use of coordinated graphics Modality principle : Learning is higher with graphics and narration 

and human narration than from graphics and printed text (Mayer, 2005b). Effect sizes in 

the range of 0.97. 

Coherence principle : Learning is better when the same information is 
not presented in more than one format (Mayer, 2005c). Effect sizes in 
the range of 1.3. 

Split attention principle : Materials should be physically and 
temporally integrated (Ayres & Sweller, 2005). 

Learner-controlled pacing Segmenting principle : Learning is greater when a multimedia 

message is presented in user-paced segments rather than as a 
continuous unit (Mayer, 2005b). Effect sizes in the range of 1.0. 

Visual annotations Signaling principle : Learning is deeper from a multimedia message 

when cues are added that highlight the organization of the essential 
material (Mayer, 2005c). Effect sizes in the range of 0.6. 

Temporal contiguity principle : Learning is deeper from a multimedia 
message when corresponding animation and narration are presented 
simultaneously rather than successively (Mayer, 2005c). Effect sizes 
in the range of 1.3. 

Use of lay language, first and Personalization principle : Learning is deeper when the words in a 
second person references, and multimedia presentation are in conversational style rather than 

use of math-specific language formal style (Mayer, 2005d). Effect sizes in the range of 1.3. 

Instruction centered around Worked example : Consists of a problem formulation, solution steps, 
worked examples and the final solution (Renkl, 2005). 

Target low-knowledge learners Prior knowledge principle : Instructional strategies that help low- 

knowledge individuals may not help or may hinder high-knowledge 
learners (Mayer, 2001). Effect sizes in the range of 0.6. 

Structure emphasizing instruction : Instruction that emphasizes the 
structural features that are relevant to the correct solution procedure 
(Renkl, 2005). Grouping solution steps by goals and methods, and 
explicitly labeling the chunks as goals and method, particularly across 
different problems with different surface features but with the same 
underlying solution structure, promotes problem solving in low- 
knowledge learners (Catrambone, 1998). 

Knowledge of results during Knowledge of results and explanatory feedback promote learning 
practice (Bangert-Drowns et al., 1991; Kluger & DeNisi, 1996). 

Explanatory feedback tailored 
to participants' selection 



Subgoal chunking and labeling, 
multiple examples. 



Complementary sources of 
information — graphics and 
audio 
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Designing Assessments For Rapid and Precision Diagnosis 

To support diagnoses of student knowledge gaps, items were developed to sample 
the following topics: (a) 12 properties of algebra, (b) transformations and related 
operations, (c) arithmetic, and (d) fractions. We used a novel item-generation process 
intended to provide precise information about students' pre-algebra knowledge. Our 
assumption is that solving a problem requires the successful application of different 
concepts across multiple steps. Table 2 shows a step-by-step derivation for solving the 
equation 7x - (3x - 2) = 38 . 

Table 2 



Item Generation Technique Based on Problem Derivation 



Step 


Equation 




Operation used from previous step to the 
current step 


0 


7x-(3x-2) = 38 


step 0: 


given, solve for x 


1 


7x + (-l)3x-(-l)2 = 38 


step 0 — > step 1: 


distributive property 


2 


7x- 3x + 2 = 38 


step 1 — > step 2: 


multiplication 


3 


x(7-3) + 2 = 38 


step 2 — >• step 3: 


factor 


4 


x(4) + 2 = 38 


step 3 — >• step 4: 


subtraction 


5 


4x + 2 = 38 


step 4 — >• step 5: 


commutative property of multiplication 


6 


4x + 2-2 = 38-2 


step 5 — > step 6: 


addition property of equality (transformation) 


7 


4x + 0 = 36 


step 6 — >• step 7: 


subtraction 


8 


4x = 36 


step 7 — »■ step 8: 


additive identity 


9 


(i)4x = (l)36 


step 8 — »■ step 9: 


mult, property of equality (transformation) 


10 


(l)x = (i)36 


step 9 — »■ step 10: 


multiplicative inverse 


11 


x = {})36 


step 10 — > step 11: 


multiplicative identity 


12 


36 

X = — 

4 


step 11 — > step 12: 


multiplication 


13 


x = 9 


step 12 — > step 13: 


division 



Each step-to-step transition requires the use of a single or small number of algebra 
properties or arithmetic. The transitions were coded to capture the algebra knowledge 
required in each transition. Note that the decomposition process yielded simpler and 
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simpler equations and terms, which we used as a source for items. Thus our items were 
generally of the form identify a valid next step instead of solve an entire equation. 

Deriving items from a single equation yields several benefits; (a) an efficient, 
simple, and precise item-generation scheme that is inherently coherent; (b) a set of items 
that students can solve quickly; (c) items amenable to a selected-response format, 
particularly the multiple true-false (MTF) format [Frisbie, 1992]; and most interestingly, 
(d) a built-in transfer test— solving the original equation (or sub-steps within the 
equation). Example items are shown in Figure 6. 

The use of the next step was inspired by intriguing findings by Kalyuga and 
Sweller (2004) and Kalyuga (2006). The authors gave students equations, but instead of 
asking students to fully solve an equation, the authors asked students to simply write 
down the first step of their solution. Kalyuga and Sweller found this technique to be 
highly predictive of performance on tasks that required students to solve the entire 
equation. High correlations were found between the two methods across different 
studies, ranging from .7 to .9. While the authors were interested in identifying ways of 
measuring expertise (with more skipped steps indicative of higher expertise), the strong 
relation between the two forms suggested a very efficient assessment. 



An equation is given beiow. For each statement beiow, circie whether the statement is T (true) or 
F (falser Circle DK (don't knowi if you are unsure. 

7.Y — (3.V— 2) = 38 (step 1) 



DP (OPER) 


72 


7.V-3.V+ 2 = 38 is a possible NEXT STEP after step 1. 


•T 


F 


DK 


CP-ADD (CE) 


73. 


7.V - (3.V - 2) = 38 has the same value as 7.v - (2 - 3.v) = 38 . 


T 


F« 


DK 


AP-ADD (CE) 


74 


7.V — (3.V — 2) = 38 has the same value as (7.v - 3.v) - 2 = 38 . 


T 


F« 


DK 


DP (CE) 


75 


7.V - 3.V -2 = 38 IS a possible NEXT STEP after step 1 . 


T 


F* 


DK 


DP (CE) 


76 


7.V - 5.V = 38 is a possible NEXT STEP after step 1 . 


T 


F* 


DK 


DP (CE) 


77 


7.V - (- 1)3.y - (- 1)2 = 38 is a possible NEXT STEP after step 1 . 


T 


F» 


DK 


DP (OPER) 


78. 


7.v + (-l)3.v -(-1)2 = 38 is a possible NEXT STEP after step 1. 


• T 


F 


DK 



DP = distributive property CE = common error 

CP-ADD = commutative property of addition OPER = operation 

AP-ADD = associative property of addition 

Figure 6. Sample multiple true-false item. Each item is mapped to a single math concept, allowing for 
precise diagnoses of math knowledge. Dots denote the correct answer. 
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Research Questions 



We had two research questions related to whether our instructional technique was 
effective: 

• To what extent can very brief exposure to instruction result in learning? 

• How effective is the instruction compared to no exposure? 

The first question addresses the basic question of whether our implementation of 
the instructional techniques can be effective. Can even very brief exposure to instruction 
on pre-algebra be effective? We know of no other work that has, as a main goal, 
explicitly designed instruction for brevity. Thus, it is an open question on whether such 
an approach could be effective. 

The second research question was comparative. If our approach demonstrated 
learning, how effective was the instruction? In our case, we used the most 
straightforward comparison— the no instruction condition. Because we were not 
manipulating instructional variables, we were very early in the research, and given that 
comparisons between instructional techniques are problematic (particularly technology 
vs. "traditional" instruction), we assumed the most appropriate and interpretable 
contrast at this stage of the research would be between students receiving our 
instruction and students receiving no instruction. 

Method 



Participants 

Background. One hundred fifteen participants were recruited from three teachers 
at an urban middle school in southern California. At the end of the first semester 
participants were drawn from two 6th-grade algebra readiness classes (n = 54) and 
three 8th-grade algebra lA classes (n = 52). The sample was split about evenly by 
gender and students ethnicity was diverse, including 31% Latino, 25% Asian or Pacific 
Islander, 17% White, and 13% African American. About three fourths of students 
reported receiving A's or B's in math, and nearly all students agreed or strongly agreed 
that they were able to understand their teacher's explanations in math class, and nearly 
all students agreed or strongly agreed that they were able to read and understand most 
of the problems and explanations in their math textbook. When asked how 
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knowledgeable they were about pre-algebra, 83% of students reported being 
moderately or very knowledgeable. 

Correlations among self-reported math grades, knowledge of pre-algebra, and 
pretest are shown in Table 3. In general, achievement variables were significantly 
related with affective variables and language variables. We interpret these relations in 
light of the student background information as suggesting that our sample was typical 
of middle-school pre-algebra students in urban settings. Appendix B contains a more 
detailed description of the sample and Appendix C contains a more detailed description 
of the measures. 

Table 3 



Descriptive Statistics and intercorrelations on Background Variables 



Variable 


n 


M 


SD 


1 


2 


3 


4 5 


6 7 


Achievement 


















1. PretesP 


113 


54.2 


15.1 


- 










2. Self-reported grades in math*’ 


109 


2.9 


5.0 


-.20 


- 








3. Self-reported pre-algebra knowledge‘s 


102 


3.0 


0.6 


.43** 


_ 


- 






Affective 


















4. interest in math‘s 


78 


2.9 


0.7 




-.07 


.22* 


- 




5. Self-concept in math** 


78 


3.1 


0.8 




-.19 


.52** 


49** 




6. Belief in "math myths"® 


78 


3.2 


0.4 


42** 


.00 


.02 


.19 .30** 


- 


Language 

7. Understand teacher's explanations® 


107 


1.6 


0.5 


_ 


.13 


-.42** 


_ 29** _ 47** 


-.19 


8. Understand problems and 
explanations in textbook® 


108 


1.6 


0.6 


_ 47** 


.25* 


-.34** 


(^4** 32** 


-.35** .56** 



‘^Maximum score = 84. '’1 = Mostly A's, 2 = Mostly B's,3= Mostly C's, 4 = Mostly D's,5 = Mostly F's. A = Not 
knowledgeable at all, 2 = Somewhat knowledgeable, 3 = Moderately knowledgeable, 4 = Very knowledgeable. <*1 = 
Disagree, 2 = Disagree somewhat, 3 = Agree somewhat, and 4 = Agree. “=1 = Strongly agree, 2 = Agree, 3 = 
Disagree, and 4 = Strongly disagree. 

Design 

A pretest, posttest two-condition waitlist design was used. Participants in both 
conditions received a pretest. Participants in the experimental condition received the 
treatment (individualized remediation) followed by a posttest while participants in the 
control condition received a posttest followed by the treatment. Participants were 
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randomly assigned to conditions in the ratio of 1 participant to the control condition for 
every 3 participants in the instruction condition. 

Measures 

Pretest scales and transfer scales. The pretest consisted of 84 items that spanned a 
range of mathematical knowledge and included simple identification of math facts (e.g., 
whether 0/3 = 0) to more complex knowledge (e.g., equations with variables on both 
sides of the equations). About 40% of the items used the multiple true-false format 
(MTF) as shown in Figure 6. MTF items are highly efficient for gathering achievement 
data and they tend to yield higher reliabilities than other selected-response formats 
(Frisbie, 1992). Appendix D contains example items from the pretest. 

Twelve scales were developed and tested in the pretest. Scales were dropped from 
the analyses for the following reasons: (a) reliability less than .60 (3 scales), (b) no 
instruction delivered that directly mapped to the scale (2 scales), (c) the scale had less 
than 4 items regardless of reliability (3 scales). 

Two complex tasks were created, one related to fractions and the other scale 
related to rational number equivalence. The fractions scale was constructed-response 
items that required students to evaluate or solve a problem related to adding or 
multiplying fractions. The rational number equivalence scale was an explanation task, 
which required students to explain how to find equivalent fractions and to provide 
numeric examples. Appendix E contains the transfer items and the scoring rubric. 

Table 4 shows the scale information for the pretest scales and the complex task 
scales. Overall, the pretest subscales retained for analyses had moderate reliability and 
the transfer scales had acceptable reliabilities. 
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Table 4 

Scale Reliability 



Scale 


n 


No. of items 


Alpha 


Pretest 


81 


84 


.89 


Pretest subscales 


Adding fractions 


101 


8 


.68 


Distributive property 


97 


8 


.70 


T ransf ormations 


104 


6 


.63 


Multiplicative identity 


93 


7 


.61 


Complex task scales 


Multiplying and adding fractions 


96 


6 


.71 


Rational number equivalence 


113 


2 


.75 



Diagnosing Knowledge Gaps 

Bayesian network (BN) of pre-algebra knowledge. We developed a BN to 
represent the domain of pre-algebra. Each hypothesis node could assume the state 
understands or does not understand, and each observable node could assume the state 
correct or incorrect. Participants' performance (correct or incorrect) on a test item was 
used as inputs to the observable nodes. Relations captured the presumed knowledge 
dependencies among the different concepts. For example, understanding the concept of 
addition implies understanding of the concept commutative property of addition. The BN 
was constructed by an electrical engineering graduate. In general, nodes higher in the 
network tended to be more conceptual and abstract (e.g., simplifying) while leaf nodes 
tended to be skill-oriented (e.g., multiplicative identity). The BN included 28 concepts 
related to the properties of algebra, arithmetic, equality, evaluation, fractions, 
simplifying, rational numbers, and factoring. 

A student understanding a concept was operationalized as a fusing of test item 
performance data across the items that required use of the particular concept. The items 
sampled a variety of conditions (e.g., identifying the "next step" in a solution process; 
use of multiple problems that incorporated fractions, integers, variables; correctly 
solving a problem). 

Given a student's performance on the items of the type shown in Figure 6, the BN 
was used to compute the probability of a student understanding various pre-algebra 
concepts. The probabilities were used to classify a student's understanding on a 
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particular concept as high, medium, or low. This classification was taken as the 
diagnosis of a student's knowledge, and used as the basis for determining what 
concepts each student would receive. 

Procedure 

A pretest was administered to students via the classroom teacher. The pretest was 
a self-contained paper booklet, with all the directions contained in the booklet. Teachers 
were instructed not to help students with the content. Students had 60 minutes to 
complete the pretest. The teachers reported that students finished the pretest between 
45 and 60 minutes. 

The following week, which contained a gap of one instructional day, students 
were administered pre-algebra instruction on the computer in a computer lab setting. 
Students were randomly assigned to computer stations. Students received 
individualized instruction in the following sense. Based on students' pretest 
performance, student received instruction that they were predicted to need help on. In 
general, students received instruction on 4 to 6 concepts within the allotted 45-minute 
period. Participants in the instruction condition received the instruction before the 
posttest, and participants in the control condition completed the posttest before 
receiving instruction. 

The posttest was tailored to each individual and included only the concepts the 
student received instruction on. The items on the posttest were identical to the items on 
the pretest. After completing the posttest, students filled out a demographic survey, a 
satisfaction survey, and a "math myths" survey. 

Results 



Preliminary Analyses 

Check for accuracy of diagnoses. Analyses were conducted to evaluate the quality 
of the classification of students into high, medium, and low knowledge categories. We 
checked for group differences on the pretest by classification. There were significant 
differences by knowledge level for the adding fractions and transformations scales, with 
students classified as low performing the lowest, followed by students classified as 
medium and high. For the distributive property scale, there was no difference between 
the medium and high groups, and for the multiplicative identity scale, there was no 
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difference between the low and medium groups. Because we were only interested in 
participants who demonstrated knowledge gaps, we collapsed the low and medium 
groups. Overall, with the collapsing, the groups appeared differentiated. 

Check for pre-existing group differences. A t test was performed to test for a 
preexisting difference in group means between the control (M = 57.7, SD - 16.1) and 
experimental (M = 53.0, SD - 14.6) conditions. The difference between the two 
conditions was close to significant (t(lll) = 1.45, p - .15). Because the Mest suggested a 
potential difference between groups on pre-algebra knowledge, subsequent between- 
group analyses included the pretest as a covariate. 

Check for implementation of the treatment. The proportion of time spent on the 
instruction relative to the actual runtime of the video instruction was computed for each 
student. The computation was based on students' log files that contained event markers 
indicating the start and end times related to the instruction. A value less than 1 suggests 
that the student had skipped through the video and a value greater than 1 suggests the 
student replayed portions of the video. 

Table 5 shows the distribution of students with respect to the time spent on 
different instruction for different concepts. As Table 5 shows, a number of students did 
skip the instruction. We used the proportion measure as an indicator of treatment 
implementation and reasoned that if students did not receive sufficient exposure to the 
instruction, then they could not profit from it. Thus, we excluded from subsequent 
analyses students who spent less than 2/3 of the time viewing the instruction. 
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Table 5 



Distribution of Students Spending Different Amounts of Time on the initial Instruction by Initial Skill 
Classification 



Concept 


Instructional 

time 

(mm:ss) 


Proportion of time spent on the initial instruction 


<0.33 


0.33- 

0.67 


0.67- 

1.00 


1.00- 

1.33 


>1.33 


Adding fractions 


10:58 


10 


5 


15 


12 


2 


Distributive property 


4:09 


12 


11 


11 


12 


2 


T r ansf or mations 


4:03 


7 


9 


14 


7 


0 


Multiplicative identity 


3:04 


4 


3 


10 


13 


2 



Main Analyses 

Overview of analyses. The main analyses addressed the two research questions: 
(a) To what extent can very brief exposure to instruction result in learning? and (b) How 
effective is the instruction compared to no exposure? 

In general, the analyses included only participants who met the following criteria: 

• Presumed treatment compliance — participants spent at least 2/3 of the 
prescribed instructional time on the instruction (see Check for implementation of 
the treatment section). 

• Presumed low or medium knowledge, based on the Bayesian network 
classification using pretest scores. 

Note that analyses were conducted by scale. For example, participants who 
received instruction on adding fractions were tested for learning by comparing posttest to 
pretest performance on the adding fractions scale. The items in the scale were identical on 
both occasions. Because participants received instruction on multiple concepts the same 
participant was included in analysis related to the concepts. For example, if a 
participant received instruction on adding fractions and multiplicative identity, then that 
participant was included both in the adding fractions analysis and in the multiplicative 
identity analysis. Thus, caution is warranted on interpreting the results as there may be 
spillover effects. 

Between-group comparisons included the pretest as a covariate, as the random 
assignment resulted in pretest performance favoring the control condition {p - .15). The 
pretest included the same items on all posttest scales, in addition to more basic math 
facts. The pretest was taken as a much broader measure of pre-algebra knowledge than 
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any one scale. Support for this interpretation is seen in the significant correlations 
between the pretest and nearly all achievement measures, attitudinal, and language 
measures (see Table 3). 

Research question 1: To what extent can very brief exposure to instruction result 
in learning? To address the first question, we examined within-subjects learning from 
pretest to posttest, by condition. Table 6 shows the results of the analyses. In the no- 
instruction condition, there were no significant differences between pretest and posttest 
on the scales or on the transfer tasks. This result strongly suggests opportunity-to-learn 
effects due to instruction and (re)testing effects were not strong enough to affect 
posttest performance. In the instruction condition, students in general performed higher 
on the posttest compared to the pretest, on nearly all the scales. Posttest results for the 
complex task are tentative. 

The within-subjects results provide evidence that our computer-based instruction 
was effective on posttest scales and the results are suggestive of a learning effect on the 
complex tasks. While the general finding that learning occurred in light of instruction is 
neither surprising nor interesting. What is remarkable is that learning occurred with 
very brief exposure to instructional activities that were apparently very effective (effect 
sizes from .50 to 1.4). 
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Table 6 



Within-Subjects Comparison of No Instruction and Instruction Conditions 











No instruction 










Instruction 










Scale 

max. 




Posttest 


Pretest 


Test of group 
differences (paired t 
test) 




Posttest 


Pretest 


Test of group 
differences (paired t 
test) 




n 


M(SD) 


M(SD) 


t 


df 


p d 


n 


M(SD) 


M(SD) 


t 


df 


p 


d 


Scale 






























Adding fractions 


8 


9 


4.67 (2.29) 


3.67 (1.66) 


1.50 


8 


.17 - 


22 


4.95 (2.13) 


3.91 (2.00) 


2.63 


21 


.02 


0.50 


Distributive property 


8 


13 


4.46 (1.45) 


4.62 (2.43) 


-0.23 


12 


.78 - 


18 


5.44 (1.50) 


4.44 (2.12) 


1.80 


17 


.09 


0.54 


T ransf ormations 


6 


13 


4.23 (1.79) 


3.62 (1.80) 


1.98 


12 


.07 0.34 


21 


4.76 (1.09) 


3.81 (1.91) 


3.21 


20 


<.01 


0.61 


Multiplicative identity 


7 


7 


5.29 (1.25) 


5.00 (1.83) 


0.60 


6 


.57 - 


21 


5.81 (0.98) 


4.00 (1.61) 


5.29 


20 


<.01 


1.36 


Complex tasks 






























Multiplying and adding 
fractions 


6 


12 


3.00 (1.28) 


2.67 (1.72) 


0.74 


11 


.47 - 


11 


4.09 (1.58) 


3.36 (2.25) 


1.62 


10 


.14 


0.37 


Rational number 
equivalence 


6 


13 


1.46 (2.18) 


1.08 (1.89) 


0.81 


12 


.43 - 


12 


4.25 (2.56) 


3.17 (2.48) 


1.78 


11 


.10 


0.43 
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Research question 2: How effective is the instruction compared to no exposure 
to instruction? To address the second question, we examined differences on the posttest 
between the instruction and no-instruction condition. Table 7 shows the results of the 
analyses. With the exception of the adding fractions scale, there were significant 
differences {p < .06) on the other posttest scales and on the transfer tasks. The effect 
sizes ranged from 0.5 to 0.9, suggesting the instruction was effective. 

The between-subjects results provide evidence that our computer-based 
instruction was effective, compared to having no instruction. Not only did participants 
learn in general as suggested in the within-subjects analysis, but the learning 
attributable to the instruction (taking differences in pre-algebra knowledge into 
account) resulted in participants in the instruction condition performing much higher 
than participants in the no-instruction condition (effect sizes range from 0.8 to 0.9). 

The most intriguing finding was an effect on the students' performance on the 
transfer tasks. The complex tasks were constructed requiring students to generate a 
solution (vs. the MTF format). In addition, the rational number equivalence task 
required participants to first explain how to find fractions that are equivalent to 2/3, 
and then provide examples. Student responses were judged for conceptual 
understanding (vs. procedural). Participants in the instruction condition outperformed 
participants in the no-instruction condition on the posttest complex tasks, with effect 
sizes of .9 (multiplying and adding fractions) and .5 (rational number equivalence). 
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Table 7 



Time on Task, Adjusted Means, Standard Errors, and Analysis of Covariance (ANCOVA) for Pre- Algebra Concepts 





Time spent on task 
components 
(instruction condition) 








Comparison of treatment effects 










instruction 


Practice 






instruction 


No instruction 




ANCOVA^ 






M(SD) 


M(SD) 


Scale 

max. 


n 


Adj. M 


SE 


n 


Adj. M 


SE 


df F ratio 


P 


effect 

size'’ 


Scale 

Adding fractions 


10:46 (2:37) 


6:30 (3:32) 


8 


22 


5.02 


0.40 


9 


4.52 


0.62 


28 


0.46 


.50 




Distributive property 


4:09 (0:57) 


1:55 (0:40) 


8 


18 


5.39 


0.36 


14 


4.22 


0.41 


29 


4.46 


.04 


0.76 


Transformations 


3:50 (0:25) 


3:14 (2:19) 


6 


21 


4.90 


0.26 


13 


4.00 


0.33 


31 


4.58 


.04 


0.77 


Multiplicative identity 


3:33 (2:21) 


2:31 (1:43) 


7 


21 


5.90 


0.21 


7 


5.03 


0.38 


25 


3.91 


.06 


0.91 


Complex tasks 

Multiplying and adding 
fractions 


19:17 (3:37) 


13:17 (6:08) 


6 


12 


3.98 


.31 


13 


2.94 


.30 


22 


5.74 


.03 


0.91 


Rational number equivalence 


- 


- 


6 


12 


4.09 


.65 


13 


1.61 


.62 


22 


7.51 


.01 


0.50 



‘^Between-groups df=l. '’Hedge's g. 
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Discussion 



This study investigated whether an integrated instructional and assessment 
system, drawing on empirically supported instructional design techniques and 
assessment approaches, could result in low prior knowledge students learning pre- 
algebra topics after very brief exposure to instructional activities — typically less than 5 
minutes of direct instruction and less than 3 minutes of practice. 

We found evidence of instructional effectiveness. Participants who received 
instruction outperformed participants who did not receive instruction on nearly all 
measures of performance. This finding held for posttest scales (that were identical to 
pretest scales) and for near transfer items that required solving fraction problems or 
required explanations related to rational number equivalence. While such differences 
are unremarkable in and of itself, what is remarkable is that such differences were 
observed given the brevity of the instructional treatment. 

Implications 

The most important finding is that we have evidence that computer-based 
"instructional parcels" can be developed to provide very brief instruction that results in 
learning of pre-algebra content. The most important implication of this finding is that 
our approach is practical and feasible, and intended to work in field settings. These 
instructional parcels were designed to be self-contained, stand-alone content with 
embedded practice. While we did not implement a fully automated system, every 
aspect of the work has been designed to support such an implementation. For example, 
the pretest items (MTF format) were paper-based but they were designed to be 
implemented on a computer and designed to work with Bayesian networks (e.g., 
dichotomous scoring simplifies the integration with Bayesian networks). 

A persistent general question that underlies our work is to what extent can 
"dumb" technology using smart methods be used to result in significant learning 
outcomes? Is the only way to meet the "2-sigma" challenge (Bloom, 1984) with 
intelligent tutoring systems (ITS)? Are there alternative instructional methods and 
technologies that can be used to replicate many of the desirable techniques associated 
with human tutors and ITSs, but in a way that does not rely on artificial intelligence 
technologies? In particular, ITSs require extremely detailed student models and 
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sophisticated inference engines. Development of ITSs is expensive, proprietary, and 
require very specific skills. 

Our approach is to use technology to perform tasks that result in student learning, 
cannot be done feasibly without technology, fills a capability gap that exists in the 
classroom or with the instructor, is simple and easy to use, and is flexible — usable 
under a variety of conditions. The combination of judicious design of multimedia 
messages, tailored feedback addressing specific misconceptions, and multiple-example 
practice opportunities, we believe, is key to promoting learning in general, and perhaps 
critical when learning is expected to occur under brief conditions. 

Next Steps 

There are several next steps of this work. First, the measures need to be refined so 
that more scales can be included in the analyses. Several scales were dropped due to 
poor reliability and low number of items. A second area is to examine retention: While 
effects were demonstrated, they were based on testing immediately following the 
intervention. Demonstrating longer term retention would bolter claims of effectiveness 
considerably. Another area to investigate is the accuracy of the initial diagnoses. While 
we used a Bayesian network to compute probabilities that a student understood 
different concepts, we did not conduct an extensive examination of how accurate the 
diagnosis was. This is clearly an area in need of further investigation, as 
individualization is in large part based on accurate diagnosis of knowledge capabilities 
and gaps. Finally, our initial objective was to design the most effective instruction by 
integrating a variety of techniques. The drawback of this approach is that it is not 
possible to identify the relative effectiveness of the various techniques. Future work that 
addresses this issue may yield even more efficient instruction by providing information 
that would inform decisions about tradeoffs among development time, development 
cost, degree of learning, and instruction and practice time. 

The signature properties of our instructional parcels is that they are brief, they 
combine assessment, instruction, and practice, and they are stand-alone and not 
intended to replace teachers or represent large curricular units, but rather are meant to 
provide instruction on a single topic. We believe our approach is feasible and applicable 
to a variety of contexts: from ADL in military training contexts to K16 contexts to 
provide supplementary help to students on an as needed basis. To realize the promise 
of individualized instruction, systems need to address the practical constraints of 
educational and training environments. Instructors in school settings typically have 
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well over 100 students and are often limited in what they can do to meet individual 
student needs— whether it is one-on-one tutoring, keeping track of students' progress, 
or even correcting homework. Flexible instructional parcels may be one solution. 
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APPENDIX A 



SUMMARY OF DESIGN GUIDELINES IMPLEMENTED IN INSTRUCTION 



Key 

instructional 

design property Implementation Multimedia principle 



Use of 
coordinated 
graphics and 
human 
narration. 



The instruction combined graphics and narration. Text was used to show 
the math principle be covered and the associated type of explanation (i.e.. 
Goal, Why, How). A narration was used to step the student through the 
instruction. 



Modality principle : Learning is higher with 
graphics and narration than from graphics and 
printed text (Mayer, 2005b). Effect sizes in the 
range of 0.97. 



Complemenfary We designed the instruction so that the each channel (visual and aural) 
sources of provided differenf information. Thus, one channel alone would be 

information— insufficient to understanding the information. 

graphics and Graphics were used to show the form of the math expression or equation, 
and narration was used to provide the ongoing explanation that was 
coordinated with the graphics display. 

All the relevant material was coordinated physically and temporally. 
Equations and their decomposition appeared adjacent to each other, and 
the narration was coordinated with the 



Coherence principle : Learning is better when 
the same information is not presented in more 
than one format (Mayer, 2005c). Effect sizes in 
the range of 1.3. 

Splif atfenfion principle : Maferials should be 
physically and temporally integrated (Ayres & 
Sweller, 2005). 



Learner- 

controlled 

pacing 



Visual 

annotations 



User controls were provided so that students could stop, replay or fast- 
forward fhrough fhe insfrucfion. Instruction was segmented into major 
sections that explained what the goal of using the math operation was, why 
the math concept was important, and how to apply the operation. 



Visual annotations were added to the instruction to explicitly point out 
what the narration was referring fo. Several cuing techniques were used: 
circling the element under discussion, highlighting or superimposing an 
image to demonstrate equivalency (see Figure 2), or arrows to denote 
procedural flow. 

Color convenfions were used such fhat fhe visual armofafions were always 
the same color and different from the math equation or expression. 



Segmenting principle : Learning is greater 
when a multimedia message is presented in 
user-paced segments rather than as a 
continuous unit (Mayer, 2005b). Effect sizes in 
the range of 1.0. 

Signaling principle : Learning is deeper from a 
multimedia message when cues are added that 
highlight the organization of fhe essential 
material (Mayer, 2005c). Effect sizes in the 
range of 0.6. 

Temporal contiguity principle : Learning is 
deeper from a multimedia message when 
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Key 

instructional 

design property Implementation Multimedia principle 



Use of lay 
language, first 
and second 
person 

references, and 
use of math- 
specific 
language 

insfrucfion 
centered around 
worked 
examples 

Target low- 
knowledge 
learners 



Subgoal 
chunking and 
labeling 



Multiple 

examples 



The visual and aural information were always synchronized in delivery. 
That is, the narration was coordinated with the graphics and cuing. The 
cuing was intended to guide students' attention to the important 
information and the narration explained the related math procedure or 
concept. 

An informal delivery sfyle was adopfed rafher than a technical style. The 
use of technical math language was minimal. Typically, the math 
terminology was introduced only after the math concept was presented in 
lay language. The presentation was of the form: " . . ., and in mafh we call 
fhis fhe commufative properly of addition." 

First and second person language was used as well. 



corresponding animation and narration are 
presented simultaneously rather than 
successively (Mayer, 2005c). Effect sizes in the 
range of 1.3. 



Personalization principle : Learning is deeper 
when the words in a multimedia presentation 
are in conversational style rather than formal 
style (Mayer, 2005d). Effect sizes in the range 
of 1.3. 



All instruction was centered around the use of worked examples thaf 
demonstrafed fhe applicafion of fhe particular math concept. The initial 
expression or equation was given, and the intermediate steps of the 
solution was shown (with use of cues) and explanatory narration. 

Our target student population was low-knowledge students. 



instruction was chunked to emphasize conceptually meaningful chunks of 
a problem's solufion or subgoals. 



Worked-out example : Consisfs of a problem 
formulafion, solution steps, and the final 
solufion (Renkl, 2005). 



Prior knowledge principle : insfrucfional 
sfrafegies thaf help low-knowledge 
individuals may not help or may hinder high- 
knowledge learners (Mayer, 2001). Effect sizes 
in the range of 0.6. 

Grouping of solufion sfeps by goals and 
methods, and explicitly labeling the chunks as 
goals and method, particularly across different 
problems with different surface features but 
with the same underlying solution structure, 
promotes problem solving in low-knowledge 
learners (Catrambone, 1998). 



instruction emphasized explicit comparison of problems, use of multiple Structure emphasizing guideline : instruction 
examples with different surface forms buf same sfrucfural form. Insfrucfion fhaf emphasizes the structural features that are 
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Key 

instructional 
design property 


Implementation 


Multimedia principle 




also directing attention to both abstract and concrete commonalities among 
the math problem examples. 


relevant with respect to the selection of the 
correct solution procedure. 


Knowledge of 
results during 
practice and 
explanatory 
feedback 
tailored to 
participants' 
selection 


The content of the feedback differed depending on learners' performance, 
if a students choice was incorrect, the feedback was 


Knowledge of results and explanatory 
feedback promote learning (Bangert-Drowns, 
Kulick, & Morgan, 1991; Kluger & DeNisi, 
1996) 
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APPENDIX B 



STUDENT BACKGROUND DETAIL 

Demographics. One hundred fifteen participants were recruited from three 
teachers at an urban middle school in southern California. The participants were drawn 
from two 6th-grade algebra readiness classes (n = 54) and three 8th-grade algebra lA 
classes (n = 52). Fifty were males and 48 females, and there were 34 Latino, 27 Asian 
American, 19 White, 14 African American, 4 biracial, and 11 participants. Of the 
remaining participants, 9 did not report grade information, 15 did not report gender 
information, and 4 participants did not report ethnicity information. 

Self-reported achievement. Ninety-seven percent of the students reported 
understanding spoken English well always or most of the time, and all students 
reported reading English well. Seventy-three percent of the students reported receiving 
A's or B's in math, 16% receiving C's, and 4% receiving D's. Ninety-eight percent of 
students reported being able to understand their teacher's explanations in math class, 
and 94% agreed that they were able to read and understand most of the problems and 
explanations in their math book. When asked how knowledgeable they were about pre- 
algebra, 83% of students reported being moderately or very knowledgeable. 

Self-reported math myths. Students held a variety of positive and negative 
notions about math. For example, 83% of students disagreed with the idea that there is 
only one way to get an answer; 93% disagreed that if a math problem took longer than 
10 minutes, it is impossible; 88% disagreed with the idea that math is too hard for most 
people to learn; 97% agreed that most math problems could be solved given enough 
time; and 97% disagreed with the idea that only geniuses are capable of understanding 
formulas and equations. On the other hand, 35% of students perceived math as just 
plugging numbers into formulas, 43% of students agreed that there is a math gene that 
some people have and others don't, and 50% of students agreed that math is mostly 
memorizing. 
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APPENDIX C 



STUDENT BACKGROUND MEASURES 

Math myths. We adopted a scale from Amdahl and Loats (1995) to measure 
participants' beliefs about math. The items in this scale were: 

• When solving problems, there is only one way to get the answer. 

• If a math problem takes more than 5 or 10 minutes, it is impossible. 

• Math is just plugging numbers into formulas. 

• There is a math gene some people have and others don't. 

• Math is hard — too hard for most people to learn. 

• Math is mostly memorizing. 

• With enough time, most math problems can be solved. 

• Only geniuses are capable of understanding formulas and equations. 

• Only geniuses are capable of creating formulas and equations. 

Participants were instructed to indicate for each item, on a 4-point Likert scale (1 = 
strongly agree, 2 = agree, 3 = disagree, and 4 = strongly disagree), how much they agreed 
with the statements about math. Cronbach's alpha was .72 (n = 94). 

Interest in mathematics. We adopted a scale from Marsh, Hau, Artelt, and 
Baumert (2006) to measure participants' interest in math. The items in this scale were: 

• When I do mathematics, I sometimes get totally absorbed. 

• Because doing mathematics is fun, I wouldn't want to give it up. 

• Mathematics is important to me personally. 

Participants were instructed to indicate for each item, on a 4-point Likert scale (1 = 
disagree, 2 = disagree somewhat, 3 = agree somewhat, and 4 = agree), how much they agreed 
with the statements about math. Cronbach's alpha was .63 (n = 104). 

Self-concept in mathematics. We adopted a scale from Marsh, Hau, Artelt, and 
Baumert (2006) to measure participants' self-concept in math. The items in this scale 
were: 
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• I get good marks in mathematics. 

• Mathematics is one of my best subjects. 

• I have always done well in mathematics. 

Participants were instructed to indicate for each item, on a 4-point Likert scale (1 = 
disagree, 2 = disagree somewhat, 3 = agree somewhat, and 4 = agree), how much they agreed 
with the statements about math. Cronbach's alpha was .80 (n = 103). 

Student survey. We administered a background survey to gather information on 
participants' age, grades in math, perceived level of knowledge in pre-algebra, and 
language. We also asked students about their experience with the software and asked 
them for comments about how the system could be improved. 
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APPENDIX D 



SAMPLE OF PRETEST ITEMS 



For each statement below, circle whether the statement is T (true) or F (false). Circle DK 
Idon't knowl if voii are unsure. 


1. 


3 4 

To compute — + — , you need to make the denominators equal to 10 
6 4 


T 


F 


DK 


2. 


12 1 ^ , 12 

— X — has the same value as 

1 12 144 


T 


F 


DK 


3. 


a X (2 X 3) has the same value as (a x 2) x 3 


T 


F 


DK 


4. 


2b 1 1 

— X — has the same value as 1 x — 
2b 2 2 


T 


F 


DK 


5. 


12(3 + 4 + 1) has the same value as 12(1 + 3 + 4) 


T 


F 


DK 


6. 


12x(-l) has the same value as -12 


T 


F 


DK 


7. 


12 2 12-2 

— has the same value as ^ 

3 3 3 


T 


F 


DK 


8. 


12a ^ ^ . 0 

7 has the same value as — 

12a' a 


T 


F 


DK 


9. 


12 2 

To compute 1° make the denominators equal to 15 


T 


F 


DK 


10. 


-5x(3x8) has the same value as (-5x3)x8 


T 


F 


DK 


11. 


1 2 X — has the same value as — x 1 2 
4 4 


T 


F 


DK 


12. 


12x3x4 has the same value as 3x12x4 


T 


F 


DK 


13. 


42 

^ has the same value as -y^x 

iu 


T 


F 


DK 



Figure 7. A sample of items related to basic math facts. 
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An equation is given below. For each statement below, circle whether the statement is T 
(true) or F (false). Circle DK (don't know! if you are unsure. 



4.Y = 36 (step 1) 



51. 


4.r 3<5 

— = — is a possible NEXT STEP after step 1. 
4 4 


T 


F 


DK 


52. 


(^)4.v = (^)36 is a possible NEXT STEP after step 1. 


T 


F 


DK 


53. 


— has the same value as x . 
4 


T 


F 


DK 


54. 


‘4'X’ 

has the same value as ( 7 ) 4 .r . 


T 


F 


DK 



Figure 8. Multiple true-false format using a single step equation. 



Two steps in solving the equation below are given. For each statement below, circle 
whether the statement is T (true) or F (false). Circle DK (don't know! if you are unsure. 

4.V = 36 (step1) 

(^)4.v = (^j36 (Step 2) 



61 


To solve the problem, you need to isolate .v . 


T 


F 


DK 


62. 


1»A = ( + )36 has the same value as.r = (-|)36 . 


T 


F 


DK 


63. 


= (t)^^ is 3 RESULT of carrying out step 2. 


T 


F 


DK 


64 


= { 7)36 has the same value as t(i)4 = (t)36 . 


T 


F 


DK 



Figure 9. Multiple true-false format involving fractions and multiple steps in the prompt. 
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APPENDIX E 



TRANSFER TEST ITEMS 



Adding and multiplying fractions transfer items 

Evaluate ^ + 

Evaluate — + — + — 

10 2 5 

Evaluate — x — 

12 4 



Evaluate — x — 



4 3 

Reduce — to its simplest form. 
32 ^ 

Find the prime factors of 56. 
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Rational number equivalence explanation transfer items 



(a) Explain how to find fractions that are equivalent to — . 



(b) Show an example of how to use your method to find three fractions that are 
2 

equivalent to — . 



Scoring rubric: 



Part a; 



Score point 


Description 


3 


The student mentions all of the following: 

• you multiply (or divide) the fraction by a/a (as long as a ^ 0)* 
or you multiply (or divide) the numerator and the 
denominator by the same # (as long as it is not 0)* 

• doing the above is the same thing as multiplying by 1 

• and so the number does not change. 

*Note: The student should not be penalized if they do not include 
the statement about the number being non-zero. 

‘Note: The student should not be penalized if they do not Include 
the statement about the number being non-zero. 


2 


The student mentions that you multiply (or divide) the top and 
bottom number by the same non-zero* number. 

‘Note: The student should not be penahzed if they do not include 
the statement about the number being non-zero. 


1 


The student unclearly expresses the idea of manipulating the top 
and bottom numbers by the same number, or by something, e.g. 
student might say “multiply the fraction by 5." Or “you do the same 
thing to the top iuid bottom.” 


0 


The student gives either no e.xplanation or an incorrect explanation. 


Part b: 



The student scores a maximum of 4 points— one for each correct equivalent fraction 
given (no credit given for rewriting 2/3) and one point for showing how they found the 
equivalent fractions (1 point total for showing any work). If more than 3 equivalent 
fractions iire given, the student does not score additional points. If no equivalent 
fractions arc given, the student scores 0 for part B. 



Page 41 of 41 



